Greptile Overview

Greptile Summary

Added support for writing _SUCCESS marker files in Parquet operations by introducing a write_success_file parameter to the write_parquet method (defaults to False). The implementation correctly handles all write modes (append, overwrite, overwrite-partitions) and creates the marker file after successful completion. The feature is well-tested with comprehensive coverage across different write modes and partition configurations.

Key Changes:

• Python API: Added write_success_file parameter to DataFrame.write_parquet()
• Rust implementation: Threaded the parameter through logical plan builder and sink infrastructure
• File creation happens after overwrite cleanup operations, ensuring proper behavior in all scenarios
• Graceful error handling with warning logs if _SUCCESS file creation fails

Minor Issue:

• Missing documentation for the new write_success_file parameter in the method's docstring

Confidence Score: 4/5

• This PR is safe to merge with one minor documentation fix needed
• The implementation is sound with proper parameter threading from Python to Rust, correct placement of _SUCCESS file creation after overwrite cleanup, comprehensive test coverage, and graceful error handling. The only issue is missing parameter documentation in the docstring, which should be added before merging.
• daft/dataframe/dataframe.py needs the write_success_file parameter documented in the docstring

Important Files Changed

Sequence Diagram

sequenceDiagram
    participant User
    participant DataFrame
    participant LogicalPlanBuilder
    participant CommitWriteSink
    participant IOClient
    participant Storage

    User->>DataFrame: write_parquet(root_dir, write_success_file=True)
    DataFrame->>LogicalPlanBuilder: write_tabular(write_success_file=True)
    LogicalPlanBuilder->>LogicalPlanBuilder: table_write() creates OutputFileInfo
    Note over LogicalPlanBuilder: OutputFileInfo stores write_success_file flag
    
    DataFrame->>DataFrame: collect() - executes plan
    DataFrame->>CommitWriteSink: finalize(states)
    
    alt Overwrite Mode
        CommitWriteSink->>IOClient: glob files in root_dir
        IOClient->>Storage: list existing files
        Storage-->>IOClient: file list
        IOClient-->>CommitWriteSink: existing files
        CommitWriteSink->>Storage: delete old files
    end
    
    alt write_success_file is true
        CommitWriteSink->>IOClient: get_source(root_uri)
        IOClient-->>CommitWriteSink: source
        CommitWriteSink->>Storage: put("/_SUCCESS", empty bytes)
        alt Success
            Storage-->>CommitWriteSink: ok
        else Failure
            CommitWriteSink->>CommitWriteSink: log::warn(error)
        end
    end
    
    CommitWriteSink-->>DataFrame: written file paths
    DataFrame-->>User: DataFrame with file paths

Codecov Report

❌ Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.41%. Comparing base (aa8add2) to head (06d895c).
⚠️ Report is 1 commits behind head on main.

@@             Coverage Diff             @@
##             main    #6090       +/-   ##
===========================================
- Coverage   72.91%   43.41%   -29.51%     
===========================================
  Files         973      909       -64     
  Lines      126196   112753    -13443     
===========================================
- Hits        92016    48948    -43068     
- Misses      34180    63805    +29625